{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- language: lang-none -->\n",
    "\n",
    "    ██████╗ ███████╗████████╗ \n",
    "    ██╔══██╗██╔════╝╚══██╔══╝ \n",
    "    ██║  ██║█████╗     ██║    \n",
    "    ██║  ██║██╔══╝     ██║    \n",
    "    ██████╔╝███████╗   ██║    \n",
    "    ╚═════╝ ╚══════╝   ╚═╝    \n",
    "    \n",
    "    Deep Learning Training Platform\n",
    "    Determined AI ©️ 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BERT PyTorch\n",
    "\n",
    "Implements the [Huggingface's run_glue example](https://github.com/huggingface/transformers/blob/v2.2.1/examples/run_glue.py) in Determined Native API.\n",
    "\n",
    "## Set up the Environment\n",
    "\n",
    "### Install the CLI\n",
    "\n",
    "Follow instructions at [docs.determined.ai/latest/install-cli.html](docs.determined.ai/latest/install-cli.html) to install the Determined command line interface. To check if Determined was installed successfully, try to import it. Then set the determined_master address to your current determined master address\n",
    "\n",
    "### Install the transformer package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sh startup-hook.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model\n",
    "\n",
    "### Set up constants\n",
    "\n",
    "There are several different models that we can run from the transformer package and we set up the necessary constants. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat constants.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the dataset\n",
    "\n",
    "To download data, we use the orginal download script `download_glue_data.py` available at [Huggingface's GLUE page](https://github.com/huggingface/transformers/blob/v2.2.1/utils/download_glue_data.py). This script is already contained in the project directory.\n",
    "\n",
    "There are two options to make this data available:\n",
    "     \n",
    " * Download the data at runtime by setting the `download_data` variable to `True` in the experiment config. \n",
    "    ```\n",
    "    config = {\"data\": {\"download_data\": True}}\n",
    "    ```\n",
    "* Download the data manually to all agents and then use bind mounts to point to the (absolute) directory where the data is loacated. \n",
    "    ```\n",
    "    config = {\"data\": {\"download_data\": False}, \n",
    "             {\"bind_mounts\": [{\"host_path\": \"<abs_data_host_path>\", \"container_path\": \"./data\"}]\n",
    "             }\n",
    "    ```\n",
    "    \n",
    "This script can be used to run BERT, XLM, XLNet, and RoBERTa on multiple GLUE tasks, such as MRPC. The full list and their median results can be found at [Huggingface's GLUE page](https://github.com/huggingface/transformers/tree/v2.2.1/examples#glue). \n",
    "\n",
    "## Define a trial class\n",
    "\n",
    "Next, we define a `BertPytorch` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat model_def.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Determined experiment\n",
    "\n",
    "Finally, we create a Determined experiment by specifying all config values and submitting the experiment via `experimental.create()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pycat trial_impl.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run trial_impl.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
